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(57) Abstract 

A method of and apparatus for removing the effects of surface and near surface image (12) storage media (14) defects from a scanned 
image using an mfraral rccoid as a nomiing controL Each pixel in a visible channel of the scanned images is divided by the corresoondine 

«n™iwiJ!5v^i?*''- ^^^^ ^ d'^^^ «>f defect in the visible channel. By 

appropnatdy altenng the gam pnor to dividing the pixel infonnation. imbalances between the visible and infrared recoids which would 
^yiT.^"^ ^u*" ^^l, " '^"^ clhninatcd. To remove defect residue, a degree of nulUng is established for each defect 

region oased on the visible and mfrarcd content in that region. In one embodimcn^ the articulation gain is multiplicativcly applied to the 
logarithm of the visible and infrared recoids. ^ -pp 
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DEFECT CHANNEL NULLING 

RELATED APPLICATION 
5 This application claims the benefit of U.S. Provisional Application No. 

60/035,763, filed January 6, 1997. 

FIELD OF THE INVENTION 
This invention relates to electronic image enhancement and recovery, and more 
10 particularly to a method and apparatus which compensate for the effects of defects of the 
media on which the image is stored. 

BACKGROUND OF THE INVENTION 
Ever since the first image of an object was captured on film, a serious problem 
15 became apparent: imperfections in and on the recording medium itself which distort and 
obscure the original image that was sought to be captured. This non-image imperfection 
problem continues to plague the field of image capture and reproduction to the present day. 
These imperfections occur in many forms including dust, scratches, fingerprints, smudges, 
and the Kke. 

20 The film industry has been concerned that the problem caused by these non-image 

imperfections may jeopardize the long term future of analog images. Notwitiistanding tiic 
significant efforts that have been made to solve the problem, it nevertheless persists. This 
is particularly true with respect to enlargements and high resolution scans. Thus, the 
problem is becoming even more acute and recognized as resolutions increase. 

25 Furthermore, multimedia applications have brought widespread attention to the problem 

,^ w ith die inc rease in fihn scanning for comput er applications 

The non-image imperfection problems occur more frequently with negatives than 
witii transparencies because (1) viewing the negative requires a light to dark ratio gain (also 
known as a "gamma") of .about two to one; whereas viewing a transparency is less 

30 demanding and (2) fihnstrips are subject to more contact than are mounted transparencies. 
Some imperfections may be present in fresh photographic fihn, for exanyle, media surface 
waves or microbubbles in the emulsion. Improper processing may also introduce 
non-image imperfection problems, for example mild reticulation or residual unbleached 
silver. Some non-image imperfection problems arc introduced through improper handling 

35 of the image media which may leave oily fingerprints and other surface contaminants or 
cause physical damage such as scratches which distort the view of the image. Improper 
storage may also leave tiie fihn further exposed to defect inducing environmental influences 
such as dust and chemicals. Archival polypropylene sleeves employed to protect negatives 
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contribute to the problem by leaving hairline surface scratches as the negatives are puUed 
out and replaced into the sleeves. As iUustrated by the above examples, non-image 
imperfections bay arise out of one or more of the following: film emulsion defects; 
improper processing; and improper handling or storage which may cause extraneous matter 
5 accumulation or scratches on the image media. 

Numerous measures have been developed in the art in an attempt to address these 
problems, particularly with respect to captured high resolution images. Generally 
spealdng. these measures are in the nature of prevention and labor intensive minimization of 
non-image defects. One approach has been that of preventing introduction of non-image 
defects in the development process itself. Expensive anti-static equipment, including spray 
machines to neutralize dust attracting charges, are employed by some photo finishers. 
Photo finishers also attempted to solve some of the problems by employing diffuse light 
source enlargers that help reduce the effects of refraction of die light due to imperfections in 
the photo finishing processes. 

Another approach used in the art involves minimizing the effects of these 
imperfections once they are present by various correction techniques, most of which are 
manual and tiius highly labor-intensive and expensive. For example, during tiie photo 
finishing process, a highly trained individual might spend a great deal of time using various 
spotting dyes and an extremely smaU spotting brush in an effort to essentially paint out tiie 
imperfections in a reproduced or printed image. Another technique is to wipe on or 
immerse die negatives in a light oU in an attempt to opticaUy fiU scratches. 

As the art has developed, various attempts have been made to automate the 
correction process, particularly witii respect to digital image systems. In such systems, 
once an imperfection has been detected, various TiU" algoritiuns are used to correct tiie 
25 i mage at the imperfection site. Nevertiieless. heuristics or huma n intervention h.c hp.n 
required to detect the imperfections with a subjective threshold. Typically die area 
identified to be corrected in this manner is much larger tiian necessary, in part due to 
application of these subjective criteria for detecting defective areas. 

Automated metiiods have also been developed for detecting imperfect areas in 
recording media, as described in German patent 2821868.0 published Nov. 22. 1979. and 
cuddcd ' 'Metiioa and Device for Detecting Reconiing and Counting of Mechanical Damage 
to Moving Bands, for Example Films." The approach discussed focuses on detemuning 
the quantity of defects and shutting down processing if tiiat measured quantity exceeds 
some predetemiined maximum level of defects. In tiiis system a source of infiared energy 
impinges upon die film medium. A scanned infrared image of the film in question is then 
taken by sensors detecting reflection of the infrared energy from the film surface. 
However, several limitations are present in the system disclosed by die patent. 
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First, its pmpose is not to correa the effects of such detected film defects present on 
the film image. Instead, the system is implemented simply to monitor the prevalence of 
these defects in an automated photographic development process whereby the process can 
be automaticaUy shut down if the defect rate exceeds a prescribed level. The optical 
5 infrared padi is a reflective one from the infrared source to the infrared sensor which is 
typicaUy different from the other sensors utilized for image processing. The infrared image 
is not recorded in registry with any other scanned images from the visual portion of the 
electromagnetic spectrum. Registry refers to the precise aUgnmcnt of two related images, 
plates, or impressions so that they are held in position relative to each other. The fact that 

10 this prior art system does not record the infrared image in registry with any other images 
taken of the film is a disadvantage which in and of itself renders it extremely difficult to 
subtract out die effect of such imperfections noted in the infrared image from similar defects 
present in the visual record of the image. 

In another prior system disclosed by the present inventor, a method which 

15 compensates for the effects of storage media defects on image data is disclosed by deriving 
from the medium separate images in the red, green, blue and infrared portions of the 
electromagnetic spectrum, each corresponding to the image stored therein. As disclosed in 
U.S. Patent No. 5,266,805 issued to the present inventor on Nov. 30, 1993, red. green, 
blue and infrared Ught is sequentially directed at one side of the fUm by means of a light 

20 source and color filter wheel Corresponding red. green, blue, and infrared images formed 
by that portion of the light being transmitted through the film are digitally captured from the 
opposite side of the film. The images are preferably captured in registry to faciUtate 
subtracting out the effect of imperfecdons at locations in the infrared record from 
corresponding locations in the red, green, and blue images. The imperfections may either 

25 substantially reduce or totally occlude the infrared light However, remaining portions of 
the xnedium having the desired image widiout such imperfections are essentially uniformly 
transmissive to infrared UghL These imperfection-free portions have variable transmissivity 
in the visual spectrum which is determined by the image developed on die film. 
Accordingly, the infrared image may serve as an indicator or map of die spatial position of 

30 these non-image imperfections on and in Uie medium, diereby allowing determination of Uie 

— location and removal of die defects so that the underiying desired image may be recovered. 

In order to remove film defects from a scanned film image, an infrared image record 
is used as a norming channel by dividing each pucel in a visible channel (red, blue, or 
green) by the corresponding pixel in the associated infrared channel. Altiiough tiiis 

35 improves die image, variations in scanner contrast across the density range of die film, 
variations in image focus, variations in illumination aperture and variations in the way 
defects respond to infrared light cause imbalances between the visible and infrared records 
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leaving defects in the resulting image afta the division. This process typically decreases 
the level of defects caused by storage of the image on a medium to within approximately 
10% of their complete elimination. Although gready improved relative to the uncorrected 
original image, die nulling of defects achieved via division with the infiared image record is 
5 not sufficient for some purposes. It is. therefore, desirable to have a method insuring that 
defect nulling approaches 100% elimination for fliese purposes. 

When a transparent film is viewed, several factors attenuate UghL The actual image 
attenuates Ught by dye absorption. In color fihn. each dye is specific to one color (red. 
green, or blue), and none of the three color dyes attenuate infimed light. Defects and 
imperfections attenuate incident Ught by bending the light outside the angle seen by die lens. 
For example, scratches, fingerprints, small flecks of dust, air bubbles, and other 
imperfections refract, or bend, light away from die lens, and thus appear as darker areas in 
the stored image. This refraction by defects is nearly die same in infrared light as in visible 
light; Uierefore. an infrared image shows die location and magnitude of defects wifliout die 
visible image. For any given pixel, if the visible measurement is divided by die infrared 
measurement, any attenuation caused by defects wiU be divided out, in tiieory leaving only 
die attenuation from die visible dye image. The dieoiy of infrared division is surprisingly 
powerful. 

In practice, however, it was found that defects did not attenuate infrared light 
exactiy the same as visible Ught. The discrepancies between theory and practice are 
attributable to several factors. In particular, the index of refraction is sUghUy different 
between infrared and visible Ught. For example, (1) some dust flecks, even diough very 
diin. may absorb some light; (2) die infrared and visible light sources in die scanner may 
not be precisely aligned causing rays bent by a defect to be "seen" somewhat differendy 
under infrared Ught tiian by v isible light; and (3) die imaging system in die scanner typically 
has less sharpness when using infrared as compared to visible light. Because of diese 
discrepancies, if die visible record is simply divided by die infiared record as in die method 
disclosed by above-referenced U.S. Patent 5.266.805. defects are gready attenuated but not 
entirely eliminated. That mediod leaves a ghost, or residue, of die defect diat stiU requires 
some type of manual intervention to fix. It is an object of die present invention to eUminate 
manual intervention in correction of defects. 

Accordingly, it is an advantage of die present invention to enhance image recovery 
and eliminate effects of media surface defects on images stored in diat media. 

To achieve this and otiier objects which will become readily apparent upon reading 
die attached disclosure and appended claims, an improved mediod of and apparatus for 
defect channel nulling which significandy reduces die effect of defects on the image 
recording medium is provided. Additional objects, advantages, and novel features of the 
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invention wiU be set forth in part in the description which foUows, and in part will become 
apparent to those skiUed in the art upon examination of the following, or may be learned by 
practice of die invention. The objects and advantages of the invention may be realized and 
attained by means of the instrumentaUties and combinations particulariy pointed out in the 
5 appended claiins. 

SUMMARY OF THE INVENTION 
A method and apparatus for nulling the effect of media defects introduced by storing 
an image on a medium. Red, blue, and green image signals are collected by passing red, 

10 blue, and green Hght through a medium containing an image and having some surface 
defects. The collected red. blue and green image signals are affected by the presence of 
inoage medium defects. A defect signal indicating die defects of the image storage medium 
is received from the same defective image storage medium. The defect signal is in register 
widi the image signal. Gain is applied to the defect signal to generate a normalized defect 

15 signal. The normalized defect signal is then subtracted from the image signal to generate a 
recovered image signal. The defect signal and recovered image signal arc compared, and 
from tills comparison a second gain is derived which is proportional to the degree tiiat tiie 
deterioration indicated by tiie defect signal is included in die image signaL This second gain 
is dien applied to die defect signal and diis how normalized defect signal is dien subtracted 

20 from die image signal to generate a recovered image signal to minunize die effects of drfects 
due to storage, handling, processing or which are inherent to die medium prior to recording 
of any image. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

J^Il.^. a schematic represent ati on of a re presentative system for implementing the 



invention 

Fig. 2 is a flow diagram indicating die overall processing sequence for defect 
channel nulling. 

Fig. 3 is a flow diagram indicating die sequence of variable initialization. 
30 Fig. 4 is a flow diagram of die pyramid building process. 

Figs. 5A, 5B, 5C and 5D are diagrams which illustrate the process of pyramid 
downsizing. 

Fig. 6 is a diagram which illustrates die process of pyramid upsizing. 
Figs. 7A, 7B, 7C, and 7D are diagrams which illustrate the process of pyramid 
35 upsizing. 

Fig. 8 is a diagram depicting die process of differencing image-related signals. 
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Figs. 9A and 9B are flow diagrams indicating the steps in the process of coirebting 
the image and defect pyiamids. 

Figs. lOA and lOB are flow diagrams indicating the steps in the process of 
correlation histogram generation. 

Rg. 1 1 is a graph oif the correlation histogram. 

Figs. 12A and 12B are flow diagrams indicating the steps in the process of 
calculating tiie correlation cutoffs. 

Fig. 13 is a flow diagram detailing tiie steps in the process of calculating the cross 
conelation and autocorrelation. 

Fig. 14 is a flow diagram indicating die steps in die process of calculating the 
clamped element 

Figs. 15A and 15B are flow diagrams indicating the steps in the process of 
calculating the correlation value. 

Rg. 16 is a flow diagram indicating the steps in the process of calculating the bound 
15 correlation value. 

Fig. 17 is a flow diagram indicating the steps in tiie process of reconstructing die 
fuUpass image. 
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DETAILED DESCRIPTION 
Fig. 1 is a schematic illustration of a representative system 10 for botii capturing a 
visible image and an infrared image from tiie frame 12of fihn 14 and storing the data in a 
suitable computer 16 for additional processing as further described hereinafter. An 
appropriate sensor, such as a color camera 18. receives image information from the frame 
12 when it is illuminated by a light source, such as visible light source 20 or infrared light 
s ource 22. The proces s of gen erating s equential i mages may of course be automat ed as 
"shown generally by the control line 24 from the computer 16 to die image generating 
components of the system 10 as is well known in the art. A keyboaixi 26 provides a 
conventional user interface to the computer 16. and a color monitor 28 is further provided 
for viewing die various images as desired. 

When implemented, the present invention, unlike the prior art, nulls defects even in 
tiie case where tiieie me discrepancies between the records. The present invention nulls, 
that IS to say eliminates by subtraction, die effects of surface and near surface defects on die 
image media. The processing computer 16 is. in effect, given not only a visible record 
containing bodi image and defect infomiation but also an infhmxi r«x)rd containing only the 
defect information. In a preferred embodiment, the infrared record further consists of a 
gain control to adjust tiie intensity of the information in die infrared record. In practice, tiie 
gain control needs to be adjusted over a somewhat limited range. This range may only need 
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to be a few percent, such as between zero and five percenL The computer also has the 
ability to adjust the gain control to achieve the best nulUng effect after the division process, 
such that none of the defect appears in the image, either as a positive ghost or as a negative 
ghost. 

An overview of how the gain is used to control the intensity of the information 
contained in the infrared record wiU be described next. This description will cover use of 
the gain in two different ways. It may be either muWpUed by the image infonnation. or the 
image information may be raised to the power of the gain. 

In the first type of use. assume that the gain is a constant, which when multiplied by 
the information in the infrared record, gives an infrared product The information in the 
visible record is then divided by the infrared product. The resulting corrected image wiU be 
made brighter or dimmer by the gain constant depending on whether the constant is less 
than or greater than one respectively. However, -using the gain constant as a simple 
multiplier will produce no resulting effect on the degree of defect nulUng since the entire 
15 corrected image wiU have been subjected to the same gain constant. 

In the second type of use. the infrared record data is raised to the power of the gain, 
which produces an indication of "intensity." In the context of nulling, a definition of 
intensity is the contrast measured (that is to say. the ratio of light to dark or gamma). The 
infrared record is raised to the power of the gain. The image data is then divided by the 
infrared data after it has been raised to the power of the gain. For example, supposing a 
particular defect absorbs 20% of the light. Without the defect, the visible signal might 
measure 50% and the infrared signal 100%. With the defect the visible signal would 
measure 50%* (100% - 20%) = 40% and the infrared signal would measure 100%* (100% 
- 20%) = 80%. To further facilitate the manipulation of the image data, the visible and 
infrared records are converted to natural logarithms. Use of logarithms is expedient 
because it not only changes the division of linear visible data by the linear infrared data to a 
subtraction of the log infrared data from the log visible data, but raising the Unear infrared 
data to the power of the gain, or gamma, becomes a multiplication of the gain by the log 
infrared data. 

As a further refinement, the gain may be adjusted separately for different spatial 
frequency bands. Tins overcomes some of the limitations in real lenses and CCD sensors 
that receive an infirared image with less focus and sharpness than a visible image. For more 
effective nulling of defects, the computer is programmed to find a higher gain matching the 
infrared record at higher frequencies to overcome this limit in real lenses and sensors. 
Without this different correction for spatial frequency, an effect is seen similar to that 
produced by an unsharp mask in which the details of a defect are attenuated less than the 
overall defect. Further, it has been found tiiat the degree of correction required tends to 
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vary between scans as the focus errors in the lens interact with film waipage. Hence, 
nulling must be perfomied on each specific image, and even on specific areas in an image, 
rather than as a rote shaipcnmg of the entire in&ared image. 

A cross correlation or autocorrelation effectively averages the power of all 
5 frequencies; however, if the niean has a non-zero magnitude, this zero fiequency mean, or 
"DC term. usuaUy predominates. Because the mean is not part of the signal of interest in 
this process, the cross correlations and autocorrelations would give false indications. The 
division of tiie visible and infiared images into frequency bands has an advantage in that 
each band has zero mean, solving the potential problem of tiie zero fiequency mean witii tiie 

10 autocorrelation and cross correlation. However, even without a nonzero mean, it is still 
possible for widely different frequencies to beat together in tiie cross correlation and 
autocorrelation terais similar to heterodynes in radio tiieory. Use of discrete frequency 
bands has tiie additional advantage of preventing widely different frequencies from 
combining in tiiis way. Frequency bands also allow die use of die logaritiimic domain to 

15 simplify the application of a gain and die division of visible pixel data by in&ared pixel data 
as a subtraction while still allowing die cross correlations and autocorrelations to be 
performed with zero mean. 

In anotiier embodiment, gain is adjusted separately for different spatial regions of 
die image. This has been found to overcome a number of limitations in real scanners tiiat 
20 illuminate and focus the edges of a film differendy than the center. Further, use of spatial 
regions overcomes problems with different kinds of defects exhibiting sUghtiy different 
attenuation characteristics between yisible and infrared light, often because of a 
misalignment between die scanner's visible and infrared illumination systems. The region 
over which Uie gain is aUowed to vary is called die "smudging" region. 

The computer is programmed to calculate nulling gain in one of several ways. In 
one^rabodiment, die coinpmer generate a mWur^^ 

by measuring the cross correlation between a candidate restored image and die defect 
image. The cross correlation is die product of die image and defect, pixel by pixel, called 
die point cross correlation, averaged over die smudging region. If die cross correlation is 
zero, tiien diat defect has been nulled. If die smudging region is too small, die process 

. °"Kht null ou t_a po rtion of t h e image diat looks like die defect along widi t he defect; 

however, if die smudging region is large enough, die chance of image detail looking 
holographically like die defect is remote, and die image is preserved as die defect is nulled. 
In anodier embodiment, die nuUing gain is calculated direcdy using die cross 
35 correlation between die uncorrected visible image divided by the autocorrelation of die 
infrared image. Fmjt, the cross correlation is averaged over die smudging region. Then die 
autocorrelation, which is the square of the value of each pixel (called the point 
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autocbirelation). is averaged over the smudging region. The average cross correlation is 
divided by the average autocorrelation to produce an autocorrelation gain for tiie region. 
When tiiis autocoirelarion gain is appUed to the infrared image and canceled from the 
visible, a corrected image is formed that gives a zero cross correlation with the infrared 
image. 

The process of the preferred embodiment will now be summarized. First, botii 
linear visible and infrared scans of die image are- taken in a conventional maniler as 
described in U.S. Patent No. 5,266.805. Pixel data from botfi scans are converted into 
logaritiimic form, and tiien each of tiiese logaritfunic images are spUt into frequency bands. 
For every corresponding frequency band, the pbcel by pucel cross correlation between the 
visible and infilled records is calculated as is the autocorrelation of tiie infrared pixel data. 
Next, the pixel by pixel cross correlations and autocorrelations are averaged, or 
"smudged." over small regions. For each region, the smudged cross correlation is divided 
by tiie smudged autocorrelation to find tiie nulling gain. The log of tire infrared bandpass 
data is then multiplied by die nulling gain, and the product is subtracted from the 
logaritiunic visible image data to obtain the corrected image data for diat bandpass. After 
this operation has been performed on all regions of the image and aU bandpasses, tiie 
corrected bandpasses are reassembled to obtain die logaritiimic data for tfie conmed image. 
The corrected image may tiien be displayed, stored, or transmitted after tiie antilog of tiie 
20 data is calculated. 

Referring now to Fig. 2, a generalized description of tiie overall process of defect 
channel nuUing is provided to facilitate its understanding. At die beginning of die process, 
die dimensions of die input image and defect data are ascertained at step 40. and progratn 
variables are initialized at step 42. Botii the image record 44 and defect record 46 are 
converted into a logaritiimic form by steps 48 and 50. respectivel y. From die log imag e 
record 52 and log defect record 54, die log image bandpass pyramid 56 and log defect 
bandpass pyramid 58 are created by steps 60 and 62, respectively. Layers of die image and 
defect bandpass pyramids 56 and 58 arc tiien correlated by step 64 to create a corrected log 
image pyramid 66. Based upon die correlation, die image pyramid 66 is corrected and 
reconstructed at step 68 to a fuUpass image 70. Finally, die corrected log image 70 is 
lA^uvcrtcd by step ii trom the logaritiimic format 70 to a corrected Knear format 74. Each 
of tfiese high level steps will now be described in detail below. 

Fig. 3 is die first of die diagrams which detaU die processes more generaUy oudined 
m Fig. 2. Rg. 3 provides a detailed description of the variable initialization of die pyramids 
mentioned in Fig.2 to begin die overall defect channel nuUing process of the present 
invention. First, die sequence of tiie array is set at 0 by step 80 to begin the counting 
process. Next, the image widdi and height are defined by step 82. As die levels of die 
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array are increased, the width and height of the image are calctilated as a function of the 
levels in the array at step 84. This process is repeated as long as the widdi and height of the 
cunent level arb not both equal to one as checked by condition 86. When the width and 
height of the image are both equivalent to one at decision step 88 thereby indicatmg a 
pyramid level consisting of a single pixel the counting of the number of levels is completed 
and is set by step 90. With this step, variable initiaKzation is complete. 

In one embodiment, the next step is to convert the image and defect records into 
logarithmic format Converting to a log format is desirable because of the pyramid buildtag 
process which faciUtates spUtting the images into frequency bands. Each layer of the 
pyramid represents a frequency band. Using the pyramid process allows a zero expected 
value, or mean, at each level and therefore eliminates the bias or "DC" term described 
earlier that differentiates a covaiiance and conelation. Because a zero mean can be reached, 
a linear format which could possibly result in a division of zero by zero (if the visible 
record is differentiated from the defect record by a division) cannot be used. Converting to 
a logarithmic scale solves this problem because division of the image record by the defect 
record is effected by a simple subtraction between equivalent levels of the image and defect 
pyramids. Thus, use of the log domain combined with the pyramid process solves 
problems both with DC bias in the conflation calculations and with effective division on 
separate frequency bands. 

When using the disclosed method to null out defects, it is helpful to be able to 
measure how the spatial frequency response of tiie defects in one record compares to die 
response in anotiier record. One way to do this is to perform a fast Fourier tr^sform on 
each record and compare diem. Based on the comparison, die attenuated frequencies of one 
record can be boosted to match the other. Anotiier expedient is to perform two^iimensional 
cross cor relation and autocor relation calculations. By spatially divi ding one by Uie otiier, 
Telative blumng iTtH^iSigTmay be found and corrected. In a preferred embodknent, wch 
image is divided into frequency bands. By perfonning the nuIUng process on each 
frequency band separately, any variations in frequency response between the images will be 
automatically compensated tiiercby boosting any attenuated frequencies. 

One way of dividing die frequency into frequency bands that allows die original 
. i ma ge to be reassembled is to c re a t e a band pass pyramid. In a bandpass pyramid, the 
image is first low pass filtered and dien subsampled at a lower resolution. The lower 
resolution is dicn upsampled to die original resolution in order to prxxJuce an image devoid 
of details that could not be represented by the lower sampling frequencies but which 
35 contains all the detaUs that could be represented by die lower frequencies. When diis 
upsampled image is subtracted from die original image, only the high fh^iuency details 
remain. The entire process is repeated on the subsampled image just discussed and die 
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process is continued recursively to generate successively lower and lower fitquency bands. 
Each band represents only those details that could not be represented by the lower 
frequency band. Because of the way that each of these bands is generated, if the lowest 
frequency band is upsized and added to the band above and the procedure is repeated 
recursively until the highest band is reached then the original image wiU be precisely 
reconstructed. Possible variations in this, process include the way in which an image is 
downsized or upsized. If a poor algorithm is used, there wiU be aliases in the bandpasses. 
niese aliases wiU cancel each other out as long as notiiing is done to each band; however, if 
the different bands are processed widi different gains, the aliases will affect the filial 
reconstructed image. Hius. the upsize and downsize chosen in a preferred embodiment are 
a compromise between minimizing aUases and conducting fast computations. Other 
downsizes and upsizes which are possible include the bicubic process used in Photoshop 
manufactured by Adobe Corp. 

Fig. 4 describes the pyramid buUding process in greater detail. IVpically, a 2,000 ( 
15 3.000 original pixel image is used and is downsized by one-half at each level.' For 
illustrative purposes, an original 8(8 image 100 is used. In the first step, the image 100 is 
downsized to a 4(4 image 102. Figs. 5A through 5D provide a detaUed description of the 
process of downsizing. They show how a 3(3 region 104 at one level of the original 8(8 
image 100 is downsized to a single pixel 51 at die next level. Each pixel in the 3(3 region 
104 is multiplied by a factor which is based upon the pixel's position within the 3(3 region 
104. In Fig. 5A, the pixel witii designation LI is given a factor of one, pixel L2 is given a 
factor of two. L3 has a factor of one. L6 has a factor of two. L7 has a factor of four, L8 
has a factor of two. LI 1 has a factor of one. L12 has a factor of two and L13 has a factcir of 
one. Each pixel and its conesponding factor arc shown in diagram 106. These factors are 
chosen so t hat Uie resulting single pbcel is a weighted average of die pixels in the other level 
based-oii-tiiepfOTuffity of die source level pixek dw^sFt^the i^ultingpixeL Next, the 
sum of tiie product of pixel values and factors is taken and divided by sbcteen, which h die 
sum of die factors so that die value for die resulting pixel is anti-aKased. The value after the 
division is die value of pixel SI which is die downsized representation of die original 3(3 
30 region 104. Use of die factors foUowed by die subsequent division by die sum of die 
. fa ct ors pe r mits each p ixel in die upsized l ev el to con t ribute a weighted portion to die value 
of die pbcel being calculated in die downsized level based on die upsized pbcel's proximity 
to the downsized pixel. 

The remaining diagrams in Figs. 5B dirough 5D arc examples of the above 
35 described process over different areas of die original 8(8 image 100. Hg. 5B illustrates die 
same 8(8 area 100 widi a different 3(3 region 108 widiin die 8(8 ai^ 100 being sampled. 
As in Fig. 5A, die pixels of Fig. 5B are multiplied by a factor based on each pixel's 
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respective position in the 3(3 region 108. The pixel with designation L3 has a factor of 
one. L4 has a factor of two. L5 has a factor of one. L8 has a factor of two. L9 has a factor 
of four, LID has a factor of two. L13 has a factor of one. L14 has a factor of two, and L15 
has a factor of one. Again, the sum of the pixel-factor products is taken and divided by 
5 sixteen, the sum of the factors. The value after the division is the value of pixel S2 which 
is the downsized representation of the 3(3 area 108. Fig. 5C illustrates the same 8(8 area 
100 of Kgs. 5A and 5B with a different 3(3 region 1 10 being sampled. As in Figs. 5A and 
5B. the pixels of Fig. 5C are multiplied by a factor based on each pixel's position in the 3(3 
region 1 10. The pixel witii designation Lll has a factor of one. L12 has a factor of two. 

10 L13 has a factor of one. L16 has a factor of two. L17 has a factor of four. L18 has a factor 
of two. L21 has a factor of one, L22 has a factor of two. and L23 has a factor of one. The 
sum of the pixel values is taken and divided by sixteen which is die sum of the factors. The 
value after the division is the value of pixel S4 which is the downsized representation of the 
3(3 area 110. Fig. 5D iUustrates the same process witii a different 3(3 region 1 12 of the 

15 8(8 area 100 being sampled. The value after the factor multiplication, summation and 
division calculations is the value of pucel S5 which is the downsized representation of the 
3(3 area. 

Returning to Fig. 4, the new downsized image 102 is then upsized to create an 
upsized image. Fig. 6 illustrates a region of a 4(4 area being upsized to an 8(8 area 118. 
There are several ways to upsize including using a sine function (sin x divided by x). 
However, the algorithm chosen for this invention addresses the calculation of a higher 
resolution pucel by looking at the pixels in the downsized image lying in four directions 
from tile higher resolution pixel. This algoritiim also has tiie advantage of being more 
e)q)edient in that it is much faster to compute tiian a sine function. 

The pyramid levels may be conceptually considered to lay over each otiicr. Each 
covers Qie same image, "but aTower level co^^ sparsely. Pixels in a 

specific level have pixels botii over and under tfiem in higher and lower levels. In addition, 
as seen in Fig. 7A tiurough 7D. pbcels also have pixels to the left, to tiie right, to tiie top. 
and to the bottom in the same and in different levels. As used herein, "over and under" wiU 
refer to the relation between different levels while "to the top" and "to the bottom" wiU refer 
— : — t o a spatial relationship within t he same image level. In the case where die value for the 
pixel to be calculated at a higher resolution has a pbcel dircctiy under it at a lower resolution, 
the lower resolution pixel is simply copied to give the higher resolution pbcel. A different 
situation occurs when tiie higher resolution pixel being calculated has lower resolution 
35 pbcels under it located to die right and to die left in die lower resolution level but not direcdy 
under die higher resolution. In this latter case, die algoritiim averages die two pixels of die 
lower level which lie to the right and left under die high resolution pixel. A third situation 
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arises when the higher resolution pixel to be calculated has lower resolution pixels under it; 
i.e., to the top and to the bottom in the lower resolution level, but not direcdy under it In 
tills case, die algorithm averages die two lower resolution pixels to give die higher 
resolution pixel. A fouidi scenario is where die higher resolution pixel being calculated 
5 does not have any pixels under it, whedier to die right, to die left, to die top or to die 
bottom, or direcdy under it Here, die higher resolution pbcel is direcdy in between die 
lower resolution pixels and die algoridun takes an average of four pixels. 

Figs. 7A through 7D describe upsizing in general and illustrate die four cases just 
described. Fig. 7A focuses on Pixel SI. Pixel SI is direcdy upsized to position U7 at 120 
10 by copying die data in SI to U7. Pixels SI and S2 are summed and divided by two to give 
value U8 equal to die average of SI and S2 at 122. SimUarly. pixels SI and S4 are 
summed and divided by two to give value U12 at 124. Pixels 81, S2, S4. and S5 are 
summed and divided by four to give die resulting average to pixel value U13 at 126. In 
Fig. 7B, die upsizing is focused on pixel S2. Pixel S2 is upsized direcdy to position U9 at 
15 128. Pixels S2 and S3 are summed and divided by two to give value UIO at 130. Pixels 
S2 and S5 are summed and divided by two to give value U14 at 132. Pixels S2, S3, S5, 
and S6 are summed and divided by four to give pixel value U15 at 134. In Fig. 70, die 
upsizing is focused on pixel S4. Pixel S4 is upsized directly to position U17 at 136. 
Pixels S4 and S5 are summed and divided by two to give value U18 at 138. Pixels S4 and 
S7 arc summed and divided by two to give value U22 at 140. Pixels S4. S5, S7, and S8 
are summed and divided by four to give pixel value U23 at 142. In Fig. 7D. die upsizing is" 
focused on pixel 85 and upsizes to a different region. Values for U19, U20, U24 and U25 
are calculated exactly as described above in connection widi Fig. 7C. 

Refening back to Fig. 4, the new upsized image (not shown), which came fix>m 
dow nsized image 102 is now differenced fipom die original image 100 to giv e die top octa ve 
bandpass image 144. Fig. 8 illustrates in dctaU die differencing process, ^chelementof 
die upsized smaller image 148 is subtracted from die corresponding pixel of die larger 
image 150 to produce an element in bandpass image 146. For example, BPl equals LI 
minus U7, and BP2 equals L2 minus U8. As shown in Fig. 4, this process is repeated 
recursively such diat each bandpass image represents a specific frequency band of die 
. origiikiJ image 100. The lowest level of the pyramid is a single pixel 152 which is not 
bandpassed because dierc is nothing from which to difference it. 

After die original image has been bandpassed, die defect image is also bandpassed 
using die same pyramid budding process as described above. As widi die image, each level 
35 of the defect pyramid represents a frequency band of die original defect After die image 
and defect records are divided into pyramid levels, each defect level is subtracted from die 
associated image level to recover an image widiout defects at diat level. Prior to subtraction 
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fiom the coiresponding image level, the defect level is multiplied by a gain chosen to match 
the defect part of the image to provide a better nuU after subtiacdon. Preferably, this gain is 
made a function of position within the defect record so that each element of the defect 
record can be adjusted with a different gain. 

Therearemanyways.offindingthisgain. One way is to iteratively try different 
gains until detail in the subtracted image record does not match detail in the defect record. 
In other words, the gain is chosen based on which one makes the image record and defect 
record as uncorrelated as possible. TTie defect channel nuUing of the present invention uses 
a non-iterative method involving direct calculation to reach an adequate degree of detaU 
uncortelation. One measurement of the detail matching is the mathetnadcalcrosscorrdation 
m which the two records are multiplied element by element To the degree both records 
match, their detail wiU track each other closely, in eidier case pnxiucing a positive result 
after the element multiplication. To the extent that they are uncorrelated, the 
crosscorrclation multipUcation will sometimes produce a positive product, but wUl just as 
likely produce a negative product Thus, when the crosscoirelation is averaged over a 
region, any nonzero value indicates a match or correlation. 

OUier refinements in gain are based on prior knowledge of record characteristics. 
For example, in one embodiment, tiie image record is set to zero whenever the defect rises 
above a predetermined intensity. Setting tiie image record to zero when the defect intensity 
IS high is an acknowledgement that beyond a certain point, an accuiate prediction based on 
tiie image is impossible, and tiie best estimate is just zero. Although it is possible to set the 
image record pyramid level to zero whenever the magnitude of the defect record exceeds a 
flireshold, the present embodiment uses instead a refinement tiiat aUows a gradual zeroing 
or blending so that no abrupt drops to zero are created in the image record. Another 
possible refmement approach includes vary ing the size of die averaging an... bi^sf d on 
image and defect characteristics. Thus, the gain refinement used by the present 
embodiment is given as an example, and is not intended to limit the process to a specific 
mode of calculating the gain. 

The next step in the defect nulUng process is to correlate the bandpass frequency 
band for botii the image and die defect pyramids. An overall description of die correlation 
l>.v.... is iUuscraiea m i^igs. 9A and 9B. The correlation process begins with the 
generation of a correlation histogram at step 156 for each corresponding level in the image 
and defect pyramids. Figs. 9 A and 9B give a detailed description of how to generate a 
correlation histogram for a single level of the image and defect pyramids. Processing starts 
with each image pixel and the corresponding defect pixel at a specified level First 
vanables used in the process are initialized at step 158. SpecificaUy. the inner limit for the 
histogram is set to zero and the outer limit is set to two. These limits are chosen to center 
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around ihe ideal situation when the correlation equals one. Because the correlation is a 
squared number, its lowest value is zero, thereby setting tiie inner limit for the histogram. 
In order for tiie approach to be balanced around one. the upper limit is set at two. 
However, it is understood that odier limits could be used. 
5 After the histogram is generated, the next step 160 is to calculate the correlation 

cutoffs which are the new upper and lower cutoff limits for the histograia 

A graph of die histogram with the correlation cutoffs is illustrated in Fig. 1 1. This 
figure shows how weU the defect record tracks the visible component of a defect The 
X-axis 162 is tiie ratio of Uie defect record to the visible record when tiie visible image has a 

10 defect The y-axis 164 is the number of pixels representing tiiat ratio weighted by tiie 
square of tiie pixels. If. for example, die visible defect exacfly tracked tiie defect record, all 
pixels would show a one to one correspondence. In practice, however, tiie infrared and 
visible records do not track exacfly. and tiie graph of Fig. 11 shows tiie degree to which 
tiiey diverge. The ten percent points 166 and 168 as illustrated attempt to show tiie range 

15 for which articulation or fine adjustment of defect gain relative to visible is expected to be 
needed. The process of calculating die correlation cutoffs is described in Figs. 12A and 
12B. These new limits 166 and 168 define tiie range witiiin which tiie values wiU be 
considered. 

After tiie cutoff values are defined, a crosscorrelation and autocorrelation are 
20 calculated as illustrated in Fig. 13. Applying the cutoff values from Fig. 12B to die image 
(also referred to as preclamping) produces a clamped image in which each individual image 
element is limited to a present value. Fig. 14 iUustrates tiie process of preclamping which is 
a way of expressing tiie importance of a particular pixel. Normally, it is expected tiiat a 
defect in tiie visible record is about 1.0 times die defect record. The purpose of nulling is to 
accommodate tiiose cases tiiat deviate from 1.0. The present embodiment preclamps tiie 
visible record To-bWeen 0.0" arfd 10 tiin^ the defect record prior to talcing the 
crossconelation and autocorrelation. In tiiis way. tiie effect of large variations in die visible 
record are taken into account but arc limited in value before tiie multiplication of tiie 
crosscorrelation and autocorrelation are performed. In Fig. 14. tiiis is done by setting tiie 
variables Low Clamp equal to 0.0 and Upper Clamp equal to 2.0. If tiie visible record 
were, for example. 4. 0 times tii^ tiefe c t r eco r d, t he preclampmg would treat tiie situation as 
tiiough it were only 2.0 times tiie defect record, but would maintain fuU weighting, which 
is matiiematically equivalent to letting it be 4.0 while only counting it witii half tiie weight. 
Anotiier weight which can be used to express confidence in a pixel is based on how far die 
calculated correction coeCBcient (defined below) varies fiom die expected value of unity for 
each individual pixel. If die variation from unity is far. tiiat pixel is given Utde weight and 
vice versa. In a specific example, any pixel that gives a single pfacel correction coefficient 
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between 0.0 and 2.0 is given fiiU weight For a conection coefficient beyond these values, 
the preclamping process effectively reduces weighting of the particular pixel proportional to 
just how far the correction coefficient is beyond diese liniits. Upon completion of the 
process described in Fig. 14. the clamped element rather than the element itself is then used 
5 to calculate the crosscorrclation and autocorrelation. 

As shown in Fig. 13, the crosscorrclation between the image and the defect leconi 
is, for a single pixel, the product of two values, the clamped element multipUed by the 
defect element The defect autocorrelation for the same pixel is the defect element value 
squared. If the crosscorrclation is tiien divided by tiie autocorrelation for tiiat single pixel, 

10 die result, in dieory, is tiie correction coefficient which is the amount of die defect record 
present in die image record. However, if tiiis approach is stricUy applied, die entire image 
will be nulled to zero for each pixel, which is obviously undesirable. Instead, only that 
component of the image tiiat matches the defect should be nulled. This is done by 
averaging tiiese values over a region. This averaging process or "smudging" is described 

15 in Fig. 9B. In one embodiment of the invention, a 3(3 element area is used for the 
averaging region because it encompasses an average full wave in die pyramid frequency. A 
larger area would give a more accurate estimate of die correction coefficient; however, 
smaller areas allow better articulation around die edges of defects which, because of 
imperfections in the angle of lighting between different colors, may require the correction 

20 coefficient to vary pixel by pixel. 

After the single element cross and autocorrelations for each element of die 3(3 
element region are calculated beginning at step 190, die nine cross correlations are averaged 
and die nine autocorrelations are averaged at step 192. As previously mentioned, diis 
averaging process across a region is called smudging. The average of die cross correlations 

^5 is termed di e regiona l crosscorrclation value. The average of die autocorrelations is die 
regional autocorrelation value. FmaUy, die regional crosscorrelation divided by die regional 
autocorrelation to give a gain measuring conrelation. For an individual pixel approaching 
zero defect where bodi die autocorrelation and aosscorrelation arc zero, litde is contributed 
by die pixel to die overaU average for its region. On die otiier hand, at a pixel Widi a strong 

30 defect record, bodi die crosscorrclation and autocorrelation are large, and die corresponding 

: pixel will contribute significandy to the average for its region. 

From the regional cross correlation and regional autocorrelation values, die 
correlation value is determined at step 194 in Fig. 9B. This process is furdier detailed in 
Figs. 15A and 15B. First an articulation gain value which is die exact gain at a pixel 

35 required to nuU die visible record is Calculated at step 196 by dividing die image element by 
die defect element This is equivalent to dividing the single element crosscorrclation, or 
image element multiplied by the defect element, by the single element autocorrelation, or 
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defect element squared. As previously described, the correlation value is obtained by 
dividing the regional crosscorrelation by the regional autocorrelation at step 198. The 
threshold mask value is obtained by dividing the regional autocorrelation by a 
prcdetemiined threshold constant at step 200. In this embodiment, the threshold constant is 
5 .015. If the autocorrelation, or defect record squared, exceeds this threshold for any pixel, 
then that pixel is aDowed to "articulate" the required correction C50cfficient for complete 
cancellation of the visible record at tiiat particular pixel, and change tiie visible image pixel 
by pixel, witiiout regional averaging. Note that if tiie threshold mask is zero at step 202, 
the new upper cutoff will be equal to die new lower cutoff which is simply equal to die 

10 correlation as shown at step 204. If die tiireshold mask is one at step 206, the new upper 
cutoff equals the old upper cutoff, originaUy assumed to be 2.0, and the new lower cutoff 
equals the old lower cutoff, originally assumed to be 0.0 as seen at step 204. Correlation is 
set to articulation at step 208 if articulation is between new upper cutoff and new lower 
cutoff, but is limited to the range between new upper cutoff and new lower cutoff. In 

15 effect, if die defect intensity is low, correlation will be left unchanged, but if the defect 
intensity is high, tiien correlation will be equal to articulation. In other words, if die defect 
intensity is high, tiien die correlation will track articulation for complete nulling of die image 
in die visible record. 

The range of correction is still limited, however, to be close to die average to 

20 prevent extreihe cancellations outside die range of probability. Using die direshold mask 
value, new upper and lower cutoff values are generated which serve to bound die 
correlation value generated above. This process is referred to in Fig. 9B as calculation of 
die bound correlation value at step 210 and is described in detail in Fig. 16. The upper 
cutoff and lower cutoff values may be made more restrictive dian die full range 0.0 to 2.0. 

25 In particular, it has been found to improve reconstruction quality when only the largest 
pyramid levels have die full range, and die smaller levels are constrained to a narrow range 
based on values of correlation in corresponding areas of higher levels. Fig. 16 illustrates 
an algoridim to set die upper cutoff and lower cutoff to be plus and minus 20% from die 
actual correlation found for die previous larger pyramid leveL 

30 As shown in Fig. 9B, this new bound correlation value is dien used to correct die 

image element at srep 212. The new correlation value is multiplied by die defect element 
and diis product is subtracted from die image element. The image element is now fully 
corrected. After each image element is corrected in the manner described, reconstruction of 
die fullpass image begins as Ulustrated in Fig. 17. Starting from the bottom of die image 

35 pyramid, each level is upsized by adding each element of die upsized level to each element 
of die level below. After all die levels have been added, die reconstruction of die log image 
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is complete. Upon completion, the antilog is taken of the log image and the resulting image 
can now be displayed, stored, or transmitted. 

WhUe this invention has been shown and described with reference to particular 
embodiments tiiereof, it will be understood by those sldUed in the ait that tiie foregoing and 
other changes in form and detail may be made tiierein without departing from the spirit and 
scope of the invention. 
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1 . A method for coirecdng defects in an image comprising: 
applying light to an image storing medium; 

receiving from the image storing medium an image signal indicating an image 
deteriorated by defects; 

applying a defect distinguishing radiation to the image storing medium; 
receiving fiom die image storing medium a defect signal, in register with the image 
signal, indicating the defects in the image; 

applying a gain to the defect signal to generate a normalized defect signal; 
differencing the image signal and nomialized defect signal to generate a recovered 
image signal; and 

adjusting the gain to minimize die defects in the recovered image signal. 

15 2. The method of claim 1 wherein the image storing medium is a substrate bearing a 
two dimensional image. 

3 . The method of claim 2 wherein the substrate bears a visible image. 

20 4. The method of claim 2 wherein the defect signal indicates the locations and 
magnitudes of substrate defects. 

5 . The method of claim 2 wherein the defect signal is received from the substrate in 
infrared light 

25 

^- UiTdefects 
in die recovered image signal comprises the step of measuring the defects in the recovered 
image signal. 

30 7. The method of claim 6 wherein the step of measuring the defects in the recovered 
— — s ignal comprises th e st ep of c a lculating the correlation between the recovered image 
signal and the defect signal. 

8. The metiiod of claim 7 wherein the step of adjusting die gain further comprises the 
35 step of altering the gain as a function of the correlation between the recovered image signal 
and the defect signal. 
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9. The method Of Claim 1 wherein the defect signal includes a pluraUty of defect signal 
frequency bands, and wherein the gain is adjusted differenay between at least two of the 
bands. 



10 



10. Hie method of claim 9 wherein one of the defect signal frequency bands comprises 
the D s Factor of the image signal. 

11. nie method of claim 9 wherein the defect signal fiequency bands are logarithmically 
spaced by a frequency incrcmendng factor. 

12. The method of claim 1 1 wherein the frequency incrementing factor is two. 

13. The method of claim 9 wherein the defect signal frequency bands are linearly 
spaced. 

14. nie method of claim 9 wherein the step of adjusting the gain to minimize the defects 
in the recovered image signal comprises the step of calculating the coirelation between the 
image signal and at least one of the defect signal frequency bands. 

20 15. The method of claim 14 wherein the step of adjusting the gain further comprises the 
step of calculating the gain as a. function of the correladon between the image signal and at 
least one of the defect signal frequency bands. 
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16. The method of claim 9 wherein the defect signal received is further characterized to 
include a plurality of defect signal regions, and wherein the gain is adjusted differendy 

--between-at least tWof the defect signaTfegions. " " ' 

17. The method of claim 16 wherein the step of adjusting the gain to minimize the 
defects in the recovered image signal comprises the step of calculating the coirelarion 

30 between the image signal and at least two of the defect signal regions. 

18. The method of claim 17 wherein the step of adjusting the gain comprises the step of 
calculating the gain as a function of the correlation between the image signal and at least two 



35 



of the defect signal regions 
19. 



The mediod of claim 18 wherein the step of calculating the gain for a selected defect 
signal region within a selected defect signal frequency band comprises the steps of 



20 



10 



wo 9801142 

PCmS97/24136 

calculating the autocorrelation of the defect signal within the selected defect signal region 
and the defect signal band, and the crosscorrelation of the defect signal within the selected 
defect signal region and the defect signal band with die image signal. 

i 20. The method of claim 19 wherein the gain within the selected defect signal region 
and the defect signal band is a function of said crosscorrelation divided by said 
autoconeladon. 

21. Themediodof claun 16 wherein the defect signal received is fmther characterized as 
comprising at least one intermediate region overlapping at least two of the defect signal 
regions, and wherein the gain is further characterized as changbg smoodily across at least 
two of the defect signal regions and the intennediate region. 

22. The method of claim 20 wherein the defect signal regions are furdier characterized 
as comprising of a plurality of points, and wherein the steps of calculating the 
autocorrelation and the crosscorrelation further includes tiie steps of calculating the point 
autocorrelations of die defect signal at the plurality of points and die point cross correlations 
between the defect signal and the registered image signal at die plurality of points, and 
averaging the point autocorrelations to obtain the autocorrelation widiin a region and 
averaging die point cross correlations to obtain die cross conflations widiin a region. 

23. The mediod of claim 22 wherein die averaging steps further include spatially 
lowpass filtering the point autocorrelations to produce a smudged autocorrelation signal 
containing smudged autoconrelation points, and spatially lowpass filtering die point cross 
correlations to produce a smudged crosscorrelation signal containing smudged 
-crosscoirdation-points: 

24. The metiiod of claim 23 wherein die adjusted gam at a selected point of die defect 
signal is a function of die corresponding smudged autocorrelation point divided by die 

30 corresponding smudged crosscorrelation point 
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25. An apparatus for correcting die effect of defects in an image comprising: 
means for applying light to die image storage medium; 

means for receiving from die image storing medium an image signal indicating an 
35 image deteriorated by defects in die image storage medium; 

means for applying a defect distinguishing radiation to die image storage medium; 
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means for receiving from the image storage medium a defect signal, in register with 
the image signal, indicating the defects in the image; 

means for applying a gain to the defect signal to generate a normalized defect signal; 

means for differencing die image signal and noimaUzed defect signal to generate a 
recovered image signal; and 

means for adjusting the gain to minimize die defects in the recovered image signal. 

26. The apparatus of claim 25 wherein the recording medium is a substrate bearing a 
two dimensional image. 



10 



15 



27. The apparatus of claim 26 wherein the substrate contains a visible 



image. 



28. The apparatus of claim 26 wherein the defect signal is received fit)m the substrate in 
infrared. 

29. The apparatus of claim 25 wherein the gain adjusting means includes means for 
measuring the defects in the recovered image signal. 

30. The apparatus of claim 29 wherein the measuring means comprises means for 
20 calculating the correlation between die recovered image signal and the defect signal. 

31. The apparatus of claim 25 wherein the gain adjusting means comprises means for 
altering the gain as a function of the correlation between the recovered image signal and the 
defect signal. 

25 

327 The apparatuTof claims wfiSrcrn~the^^^ for 
measuring the defects in die image signal. 

33. The apparatus of claim 32 wherein the measuring means comprises means for 
30 calculating the correlation between die image signal and die defect signal. 

34. The apparatus of claim 33 wherein die gain adjusting means comprises means for 
calculating die gain as a function of die correlation between die image signal and the defect 
signal. 

35 
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35. The apparatus of claim 25 wherein the defect signal comprises a plurality of defect 
signal fipequency bands, and wherein the gain is adjusted differently between at least two of 
the bands. 

36. The apparatus of claim 35 wherein one of the defect signal frequency bands is a 
mean. 



10 



15 



20 



37. The apparatus of claim 35 wherein the defect signal frequency bands 
logarithmically spaced by a frequency incrementing factor. 



are 



38. n»e apparatus of claim 37 wherein the fiequency incrementing factor 



IS two. 



39. The apparatus of claim 35 wherein the defect signal frequency bands are linearly 
spaced. 

40. The apparatus of claim 35 wherein the gain adjusting means further comprises 
means for calculating the correlation between the image signal and at least one of the defect 
signal frequency bands. 

41. The apparatus of claim 40 wherein the gain adjusting means funher comprises 
means for calculating die gain as a function of the coirelation between the image signal and 
at least one of the defect signal fiequency bands. 

42. The apparatus of claim 35 wherein the defect signal is further characterized by 
25 comprising a plurality of defect signal regions, and wherein the gain is adjusted differently 

J>etW£en-atleast-two0fAe^egionsr 



43. A metiiod of processing visual image data with non-image defect data to remove 
non-image defect information from the visual image data, the non-image defect data being 
30 registered with the visual image data, comprising: 

— ■ creating, from the visual i mage d ata, a p l urali ty o f vis u al image data set s for 

respective spatial fiequency bands; 

creating ftom the visual image data a pluraUty of non-image defect data sets for 

respective spatial frequency bands substantially similar to the spatial frequency bands for 
35 the visual imaging data; 
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cross correlating the visual image data set and the non-image defe« data sets for 
respective spatial fiequency bands to obtain oossconelation variables for each of the spatial 
frequency bands; 

autocoirelating the non-image defect data set for each of the spatial frequency bands 
► toobtainautoconelationvariablesforeachofthespatialfiequencybands: 

processing the crosscorrelation variables over a pluiaUty of spatial regions to obtain 
spatially defined crosscoirclarion variables for each of the spatial frequency bands; 

processing die autocorrelation variables over a plurality of spatfd regions 

substantiaUy similar to the spatialrcgions of tiKcrosscorrcbtion variables to obtain sp«^ 
defined autocorrelation variables for each of the spatial frequency bands; 

processing the spatially defined crosscorrelation variables with the spatiafly defined 
autocorrelation variables to obtain spatially defined nulUng gains for each of the spatial 
frequency bands; 

applying tiie spatially defined nulling gains to the respective non-image defect data 
sets to obtain nulling data sets for the respective spatial frequency bands; 

applying the nulling data sets to tiie visual image data sets for the respective spatial 
frequency bands to obtain corrected visual image data sets for the respective spatial 
frequency bands; and 

merging die corrected visual image data sets for the respective spatial frequency 
20 bands to obtain corrected visual image data. 

44. A metiiod of processing visual image data with non-image defect data to remove 
non-image defect information from die visual image data, die non-image defect data being 
registered widi die visual image data, comprising: 

comparing die visual image data with the non-image defect data to determine a 
nulling gainr ' "~ 

applying die nulling gain to die non-image defect data to create nulling data; and 
applying the nulling data to die visual image data to remove non-image defect 
information therefrom. 
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_45^ 



35 



T h e m e diod of claim 4 3 f urt he r com pi is i ag identifying a plurality of spatial 
frequency bands to overcome limitations in real optics and image sensors, die visual image 
data further comprising a plurality of respective visual image data sets for the spatial 
frequency bands, die non-image defect data further comprising a plurality of respective 
non-image defect data sets for the spatial frequency bands, the nulling gain furdier 
comprising a plurality of respective nulling gain sets for die spatial fr^equency bands, and 
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the nulling data further comprising a plurality of respective nulUng data sets for the spatial 
frequency bands. 

46. The method of claim 43 further comprising identifying a plurality of spatial regions 
5 of the image, wherein the nulling gain is determined for each spatial region. 
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